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LARGE-AREA IMAGING BY STITCHING WITH ARRAY MICROSCOPE 



BACKGROUND OF THE INVENTION 

5 Field of the Invention 

[0001] This invention is related in general to the field of 
microscopy. In particular, it relates to array microscopes and 
to a novel approach for acquiring multiple sets of image tiles of 
a large sample area using an array microscope and subsequently 
10 combining them to form a good-quality high-resolution composite 
image . 

Description of the Related Art 

[0002] Typical microscope objectives suffer from the inherent 
15 limitation of only being capable of imaging either a relatively 
large area with low resolution or, conversely, a small area with 
high resolution. Therefore, imaging large areas with high 
resolution is problematic in conventional microscopy and this 
limitation has been particularly significant in the field of 
20 biological microscopy, where relatively large samples (in the 

order of 20 mm X 50 mm, for example) need to be imaged with very 
high resolution. Multi-element lenses with a large field of view 
and a high numerical aperture are available in the field of 
lithography, but their cost is prohibitive and their use is 
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impractical for biological applications because of the bulk and 
weight associated with such lenses. 

[0003] A recent innovation in the field of light microscopy 
5 provides a solution to this problem using an array microscope. 
As described in commonly owned PCT/US02/08286, herein 
incorporated by reference, an array microscope consists of an 
array of miniaturized microscopes wherein each includes a 
plurality of optical elements individually positioned with 

10 respect to a corresponding image plane and configured to image 
respective sections of the sample object. The array further 
includes a plurality of image sensors corresponding to respective 
optical elements and configured to capture image signals from 
respective portions of the object. The absolute magnification in 

15 an array microscope is greater than one, which means that it is 

not possible to image the entire object surface at once even when 
it is equal to or smaller than the size of the array. Rather, 
the imaged portions of the object are necessarily interspaced in 
checkerboard fashion with parts of the object that are not 

20 imaged. Accordingly, the array microscope was designed in 

conjunction with the concept of linear object scanning, where the 
object is moved relative to the array microscope and data are 
acquired continuously from a collection of linear detectors. 
Data swaths obtained from individual optical systems are then 
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concatenated to form the composite image of the object. 

[0004] In such an array microscope, a linear array of 
miniaturized microscopes is preferably provided with adjacent 
5 fields of view that span across a first dimension of the object 
and the object is translated past the fields of view across a 
second dimension to image the entire object. Because each 
miniaturized microscope is larger than its field of view (having 
respective diameters of about 1 . 8 mm and 200 pm, for example), 

10 the individual microscopes of the imaging array are staggered in 
the direction of scanning so that their relatively smaller fields 
of view are offset over the second dimension but aligned over the 
first dimension. The axial position of the array with respect to 
the sample object is preferably adjusted to ensure that all parts 

15 of the sample surface are imaged in a best-focus position. Thus, 
the detector array provides an effectively continuous linear 
coverage along the first dimension which eliminates the need for 
mechanical translation of the microscope in that direction, 
providing a highly advantageous increase in imaging speed by 

20 permitting complete coverage of the sample surface with a single 
scanning pass along the second dimension. Such miniaturized 
microscopes are capable of imaging with very high resolution. 
Thus, large areas are imaged without size limitation and with the 
very high resolution afforded by the miniaturized microscopes. 
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[0005] In a similar effort to provide a solution to the challenge 
of imaging large areas with high magnification, U.S. Patent No. 
6,320,174 (Tafas et al.) describes a system wherein an array of 
optical elements is used to acquire multiple sets of checkerboard 
5 images that are then combined to form a composite image of the 

sample surface. The sample stage is moved in stepwise fashion in 
relation to the array of microscopes (so called "step-and-repeat " 
mode of acquisition) and the position of the sample corresponding 
to each data-acquisition frame is recorded. The various image 

10 tiles are then combined in some fashion to provide the object's 
image. The patent does not provide any teaching regarding the 
way such multiple sets of checkerboard images may be combined to 
produce a high-quality high-resolution composite image. In fact, 
while stitching techniques are well known and used routinely to 

15 successfully combine individual image tiles, the combination of 
checkerboard images presents novel and unique problems that 
cannot be solved simply by the application of known stitching 
techniques . 

20 [0006] For example, physical differences in the structures of 
individual miniaturized objectives and tolerances in the 
precision with which the array of microscopes is assembled 
necessarily produce misalignments with respect to a common 
coordinate reference. Moreover, optical aberrations and 
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especially distortion and chromatic aberrations, as well as 
spectral response and gain/offset properties, are certain to vary 
from microscope to microscope, thereby producing a checkerboard 
of images of non-uniform quality and characteristics. Therefore, 
5 the subsequent stitching by conventional means of multiple 
checkerboards of image tiles acquired during a scan cannot 
produce a high-resolution composite image that precisely and 
seamlessly represents the sample surface. For instance, as 
illustrated in Fig. 1, assume that an array microscope includes 

10 only two miniaturized microscopes and that the second microscope 
introduces a slight rotation and offset in the image 10 acquired 
from the sample surface 12 with respect to the image 14 acquired 
by the first microscope (the dashed line 16 represents a 
perfectly aligned image) . Accordingly, the acquisition of the 

15 first frame of image tiles would produce a pattern similar to 
that illustrated in the figure. The acquisition of the second 
frame of image tiles would produce a similarly misaligned set of 
images 10 1 and 14', as illustrated. 

20 [0007] If conventional stitching procedures are used to combine 
the various image tiles, such as described in U.S. Patents No. 
5,991,461 and No. 6,185,315, the stitching of images 10 and 10 1 
will produce a seamless image of uniform quality accurately 
representing the corresponding section of the sample surface 12. 
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This is because both images 10 and 10' result from data acquired 
with the same miniaturized microscope and the same spectral 
response, gain, offset, distortion and chromatic aberrations (to 
the extent they have not been removed by correction) apply to 
both images, thereby producing a composite image of uniform 
quality. Inasmuch as stitching procedures exist that are capable 
of correcting misalignments between adjacent image tiles, a 
similar result could be obtained by stitching images 14 and 14 f , 
but the process of combining images 10 with 10' and 14 with 14 1 
would necessarily consist of separate computational phases 
wherein each pair of images is combined. The combination of 
images acquired with different microscopes, though, could not be 
carried out meaningfully with conventional stitching techniques. 
Combining image 10 1 with image 14, for example, may be possible 
as far as misalignments and offsets are concerned, but the 
combined difference could still be non-uniform with respect to 
spectral response, gain, offset, and distortion or chromatic 
aberrations (depending on the characteristics of each 
miniaturized microscope) . Therefore, the overall composite image 
could represent a meaningless assembly of incompatible image 
tiles that are incapable of producing an integrated result (like 
combining apples and oranges) . 

[0008] Thus, the prior art does not provide a practical approach 
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to the very desirable objective of imaging a large area with an 
array microscope in sequential steps to produce checkerboards of 
images that can later be combined in a single operation simply by 
aligning any pair of adjacent image tiles. Similarly, the prior 
5 art does not provide a solution to the same problem of image non- 
uniformity produced by an array microscope that is scanned 
linearly over a large area of the sample surface to produce image 
swaths that are later combined to form a composite image. This 
invention provides a general and efficient solution toward this 
10 end. 
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BRIEF SUMMARY OF THE INVENTION 



[0009] In view of the foregoing, the invention is described with 
reference to an array microscope operating in step-and-repeat 
scanning mode, but it is equally applicable to every situation 
where an array microscope is used to generate images of portions 
of a large sample area to be subsequently combined to image the 
whole area. Thus, the imaging apparatus consists of multiple 
optical systems arranged into an array capable of simultaneously 
imaging a portion of an object in a manner similar to the linear 
scanning array microscope described in PCT/US02/08286 . Instead 
of scanning the object in linear fashion, a step-and-repeat 
approach is followed and multiple sets of checkerboard images are 
generated by the two-dimensional array of miniaturized 
microscopes. By combining these multiple sets of images of the 
object taken at specific spatial intervals, a larger area than 
the field of view of the individual optical systems can be 
imaged. 

[0010] In order to enable the stitching of the various multi- 
image frames acquired during a scan in a seamless manner to 
compose a large-area image with uniform and significant features, 
the performance of each microscope is normalized to the same 
reference base for each relevant optical-system property. 



8 



Specifically, a correction-factor matrix is developed through 
calibration to equalize the spectral response measured at each 
detector; to similarly balance the gains and offsets of the 
detector/light-sources associated with the various objectives; to 
5 correct for geometric misalignments between microscopes; and to 
correct distortion, chromatic, and other aberrations in each 
objective . 

[0011] Thus, by applying the resulting correction-factor matrix 
10 to the data acquired by scanning the sample object in step-and- 
repeat fashion, the resulting checkerboard images are normalized 
to a uniform basis so that they can be concatenated or combined 
by stitching without further processing. As a result of this 
normalization process, the concatenation or stitching operation 
15 can be advantageously performed rapidly and accurately for the 
entire composite image simply by aligning pairs of adjacent 
images from the image checkerboards acquired during the scan. A 
single pair of images from each pair of checkerboards is 
sufficient because the remaining images are automatically aligned 
20 as well to produce a uniform result by virtue of their fixed 
spatial position within the checkerboard. 

[0012] Various other purposes and advantages of the invention 
will become clear from its description in the specification that 



9 



follows and from the novel features particularly pointed out in 
the appended claims. Therefore, to the accomplishment of the 
objectives described above, this invention consists of the 
features hereinafter illustrated in the drawings, fully described 
5 in the detailed description of the preferred embodiment and 

particularly pointed out in the claims. However, such drawings 
and description disclose but one of the various ways in which the 
invention may be practiced. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



[0013] Fig. 1 is a simplified schematic representation of the 
images produced by a two-microscope array at two acquisition 
5 frames taken after the sample is moved by a predetermined amount 
designed to cover a sample area larger than the field of view of 
the array microscope. The figure illustrates a physical 
misalignment in the images produced by the first and second 
microscope . 

10 

[0014] Fig. 2 illustrates the basic configuration of an array 
microscope composed of several individual optical elements formed 
on rectangular grids aligned along common optical axes. 

15 [0015] Fig. 3 illustrates a large area of an object surface 
covered by the field of view of an objective scanning in 
overlapping fashion through a step-and-repeat process. 

[0016] Fig. 4 illustrates the large area of Fig. 3 covered by the 
20 fields of view of individual objectives of a four-element array 
microscope at an initial scanning position. 

[0017] Fig. 5 is the image produced by scanning a sample area 
with an array microscope in step-and-repeat mode without 
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correction for distortion. 



[0018] Fig. 6 is an enlarged view of a section in the overlap 
region between adjacent tiles that clearly illustrates the 
5 distortion error of Fig. 5, as exhibited by the vertical 
discontinuities in the data. 

[0019] Fig. 7 is the image produced by scanning the same sample 
of Fig. 5 without correction for geometric misalignments between 
10 the two microscopes. 

[0020] Fig. 8 is the image produced with the same sample data 
after calibration of the array microscope for geometric 
uniformity, as described, and the subsequent application of the 
15 resulting correction factors to the raw image data. 

[0021] Fig. 9 is an image produced by scanning the same sample of 
Fig. 5 without correction for spectral-response uniformity 
between the two microscopes. 

20 

[0022] Fig. 10 is the image produced with the same data of Fig. 7 
after calibration of the array for spectral-response uniformity 
according to the invention and the subsequent application of the 
resulting correction factors to the raw image data. 
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[0023] Fig. 11 is an image produced by scanning a sample area 
with an array microscope in step-and-repeat mode without 
correction for differences in detector gain and offset. 

[0024] Fig. 12 is an image produced with the same data of Fig. 11 
after calibration of the array for gain and offset uniformity 
according to the invention and the subsequent application of the 
resulting correction coefficients to the raw image data. 
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DESCRIPTION OF THE PREFERRED EMBODIMENTS OF THE INVENTION 



[0025] The invention was motivated by the realization that the 
images produced by step-and-repeat data acquisition using an 
5 array microscope cannot be combined directly to produce a uniform 
composite image because of the unavoidable data incompatibilities 
produced by discrepancies in the optical properties of the 
various miniaturized microscopes in the array. The heart of the 
invention lies in the idea of normalizing such optical properties 

10 to a common basis, so that functionally the array of microscopes 
performs, can be viewed, and can be treated as a single optical 
device of uniform characteristics. As a result, each set of 
multiple checkerboard images produced simultaneously at each 
scanning step can be viewed and treated as a single image that 

15 can be aligned and stitched in conventional manner with other 
sets in a single operation to produce the composite image of a 
large area. 

[0026] As development of the invention progressed, it became 
20 apparent that the same advantages provided by it may be used when 
an array microscope is utilized with linear scanning and image 
swaths are similarly concatenated or stitched together. 
Therefore, the term "checkerboard" is used herein primarily, in 
relation to step-and-repeat scanning, to refer to image frames 
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corresponding to portions of the sample object interspaced in 
checkerboard fashion with parts of the object that are not 
imaged. Checkerboard is also intended to refer, with reference 
to linear scanning, to the collection of image swaths produced by 
5 the array detector during a scan. The term "microscope" is used 
with reference to both the array microscope and the individual 
miniaturized microscopes within the array, and it is assumed that 
the distinction will be apparent to those skilled in the art from 
the context of the description. The term "field of view" is 

10 similarly applied to both. The term "axial" is intended to refer 
to the direction of the optical axis of the array microscope used 
for the invention. The term "stitching" refers to any 
conventional procedure used to combine separate data sets 
corresponding to adjacent sections of a sample surface. "Step- 

15 and-repeat" is used to refer to a data acquisition mode wherein 
frames of data (corresponding to either a single continuous 
section or to separate checkerboard sections of a sample surface) 
are taken during a scan by translating the object or the 
microscope in stepwise fashion and by acquiring data statically 

20 between steps. With reference to data acquisition, the term 
"frame" is used to refer to the simultaneously acquired data 
obtained at any time when the system's sensor or sensors operate 
to acquire data. The term "tile" refers both to the portion of 
the sample surface imaged by a single miniaturized microscope in 
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an array and to the image so produced. The term "concatenation" 
refers to the process of joining the images of adjacent portions 
of the sample surface acquired without f ield-of-view overlaps, 
based only on knowledge of the spatial position of each frame in 
5 relation to a reference system and the assumption that such 

knowledge is correct. The term "stitching" refers to the process 
of joining the images of adjacent portions of the sample surface 
acquired with f ield-of-view overlaps, wherein the knowledge of 
the spatial position of each frame is used only as an 
10 approximation and such overlaps are used to precisely align the 
images to form a seamless composite. 

[0027] Moreover, the terms "geometric alignment," "geometric 
calibration" and "geometric correction" are used herein with 

15 reference to linear (in x, y or z) and/or angular alignment 

between image tiles produced by an array microscope (distortion) , 
and to chromatic aberration associated with the microscopes in 
the array. The term "spectral response" refers to the signals 
registered by the detectors in response to the light received 

20 from the imaging process. Finally, the terms "gain" and "offset" 
variations are used with reference to differences in the 
electrical response measured at different pixels or on average at 
diff erent , detectors as a function of variations in the current 
supplied to the light sources, in the background light received 



16 



by each microscope, in the properties of the optical systems, in 
the detector pixel responses, in the temperature of the sensors, 
and in any other factor that may affect gain and offset in an 
optical /electronic system. 

5 

[0028] Referring to the drawings, wherein like reference numerals 
and symbols are used throughout to designate like parts, Fig. 2 
illustrates the basic configuration of an array microscope 20 
composed of several individual optical elements (22a, 24a, 26a) 

10 formed on rectangular plates 22,24,26 (nine elements are 
illustrated in each grid in the figure) . Each microscope 
includes three lenses distributed along an optical axis passing 
through the plates 22, 24 and 26. These plates are arranged into 
a stack 28 aligned with another plate 30 containing area 

15 detectors 30a positioned in the image plane of the object surface 
12 to be measured. Each area detector 30a consists of many 
individual sensing elements (pixels) that are preferably 
implemented with charged-coupled-device (CCD) or complementary- 
metal-oxide-semiconductor (CMOS) arrays, or any other suitable 

20 image sensing device. 

[0029] As illustrated, this embodiment is characterized by a one- 
to-one correspondence between each optical system and an area 
detector. Thus, the field of view of each detector projected 
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through the optical system yields a rectangular image (as shaped 
by the detector) received in the image plane of the object. At 
each given acquisition time, the individual optical systems 
(objectives) of the array form the image of a portion of the 
object surface on the corresponding detector. These images are 
then read out by suitable electronic circuitry (not shown) either 
simultaneously or in series. 

[0030] Fig. 3 illustrates a strategy that can be used 
advantageously to acquire high-resolution images of an object 
area 32 larger than the field of view covered by the microscope 
objective. As mentioned above, when greater-than-one absolute 
magnification is involved, it is not possible to image the entire 
object surface at once because of the physical constraints 
imposed by the size of the objectives. Therefore, it is 
necessary to repeat the acquisition of data at locations of the 
object that have not been previously imaged. The various tiles 
34 of data are then combined in a concatenating or a stitching 
operation that requires a large number of images to be taken and 
stored. When done conventionally, the process takes significant 
time because for each data frame the object must be positioned in 
front of the optical system and often also focused before 
acquisition. 
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[0031] Such repetitive procedure is not practical (or sometimes 
even possible) with conventional microscopes because of their 
size. The typical field of view of a 40* microscopic objective 
is about 300 \im with a lens about 20 mm in diameter. Therefore, 
5 even an array of conventional microscopes (such as described in 
U.S. Patent No. 6,320,174) could not image more than a single 
tile at a time on an object surface of about 20x20 mm or less in 
size. By comparison, the field of view of each individual 
optical system in an array of miniaturized microscopes is 

10 comparable in size (i.e., about 200 pm) , but the distance between 
optical systems can be as small as 1.5 mm. Thus, the diameter to 
f ield-of-view ratio in array microscopes is about 7.5 while in 
conventional optical microscopes it is in the order of about 65. 
As a result, array microscopes are most suitable for acquiring 

15 simultaneously multiple images of portions of the sample object 
in checkerboard fashion. Because imaging by the various 
miniaturized objectives is performed in parallel, multiple tiles 
are imaged at the same time at each scanning step. This can be 
done by translating the array microscope with respect to the 

20 object (or vice versa) on some predetermined step pattern. 

[0032] For example, the areas 36 shaded in gray in Fig. 4 would 
correspond to image tiles of corresponding portions of the object 
surface 32 assuming that the array microscope had four optical 
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systems with fields of view spaced apart on the object plane by a 
distance substantially equal to the field of view of each 
individual microscope (in this example a distance to field-of- 
view ratio of about 2 is assumed to simplify the illustration) . 
In fact, a smaller distance is often required in practice to 
allow sufficient overlaps (the cross-hatched areas 38, magnified 
in the figure for clarity of illustration) during the acquisition 
steps to account for scanning misalignments and/or to acquire 
duplicate data that facilitate the subsequent stitching of 
adjacent tiles. While a typical step-and-repeat procedure with a 
single microscope requires as many acquisition frames as the 
number of tiles 34 necessary to cover the overall imaged area 32, 
the use of an array microscope makes it possible to reduce the 
frames by a factor equal to the number of objectives aggregated 
in the array. In the example shown in Figs. 3 and 4, a total of 
sixteen individual tiles would be necessary to reconstruct the 
entire object surface. Thus, using a conventional microscope, 
sixteen acquisitions would be required. Using an array 
microscope with four optical systems, it is instead possible to 
acquire four tiles simultaneously at each step and the entire 
surface can be imaged in just four steps. In the first step, the 
images corresponding to fields of view Sll, S13, S31 and S33 are 
acquired; followed by S21, S41, S23 and S43; then S22, S42, S24 
and S44; and finally S12, S32, S14 and S34. The four 
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checkerboard frames of images so acquired are then concatenated 
or stitched together to form the composite image of the object 
surface. 



5 [0033] As described, in order to enable the composition of a 

seamless, meaningful image of the large area for which data have 
been acquired, the system is calibrated according to the 
invention and the results of calibration are applied to the 
frames of data prior to stitching or concatenation. According to 

10 one aspect of the invention, the device is first calibrated to 
establish the relative position and the magnification (pixel 
spacing) of each field of view at imaging color bands (RGB) and 
corrective factors are then applied to align all image tiles 
(with respect to a fixed coordinate system) and to produce 

15 uniform magnification across the array microscope. That is, the 
system is corrected for aberrations commonly referred to in the 
art as distortion and chromatic aberration. Such calibration may 
be accomplished, for example, using prior knowledge about the 
geometry of the system or using standard correlation methods. In 

20 the former case, each tile's image is reconstructed, if 
necessary, according to such prior knowledge by applying 
geometric transformations (such as rotation, scaling, and/or 
compensation for distortion) designed to correct physical non- 
uniformities between objectives and optical aberrations within 
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each objective. The images are then concatenated or stitched to 
create a composite image. 

[0034] Because of the simultaneous acquisition of each 
5 checkerboard set of images (Sll, S13, S31 and S33, for example), 
the geometric relationship between individual optical systems in 
the array is preserved between acquisition frames. Therefore, 
this fixed relationship can be used advantageously to materially 
speed up the image combination process. Since the physical 

10 relationship between checkerboard images does not change between 
frames, once normalized according to the invention, the sequence 
of frames can be concatenated or stitched directly without 
further processing subject only to alignment to correct scanning 
positioning errors. Thus, using conventional stitching methods 

15 to seamlessly join two adjacent tile images acquired in 

consecutive steps (Sll and S21, for example), the rest of the 
tile images (S13,S31,S33 and S23,S41,S43) can be placed directly 
in the composite image simply by retaining their relative 
positions with respect to Sll and S21, respectively. 

20 

[0035] As part of the calibration procedure, this relationship 
can be established by imaging a reference surface or target 
through which the position and orientation of each field of view 
can be uniquely and accurately identified. One such reference 
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surface could be, for example, a flat glass slide with a pattern 
of precisely positioned crosses on a rectangular grid that 
includes a linear ruler with an accurate scale. Such a reference 
target can be easily produced using conventional lithography 
processes with an accuracy of 0.1 jam or better. Using a large 
number of individual target points for the calibration procedure 
can further increase the accuracy. 

[0036] The lateral position, angular orientation, and distortion 
of each optical system and detector can be accurately measured by 
determining the positions of reference marks (such as points on 
the crosses) within the field of view of each image and by 
comparing that information with the corresponding positions of 
those marks in the reference surface based on the ruler imprinted 
on it. The differences are converted in conventional manner to 
correction factors that can then be used to correct image errors 
due to the geometric characteristics of the array microscope. As 
a result, linear and angular misalignment of the various fields 
of view in the array can be corrected to establish the exact 
position of each tile within the overall composite image. Once 
so established, such correction factors can be incorporated in 
firmware to increase the processing speed of the optical system. 
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[0037] Alternatively, correlation methods can be used that rely 
only on an approximate knowledge about the position of each 
individual image in the checkerboard of fields of view. Using 
these techniques, the exact position of each tile is established 
5 by matching two overlapping sections of images of adjacent 

portions of the object (taken at different frames) . This can be 
done in known manner using, for instance, a maximum cross- 
correlation algorithm such as described by Wyant and Schmit in 
"Large Field of View, High Spatial Resolution, Surface 
10 Measurements," Int. J. Mach. Tools Manufact. 38, Nos. 5-6, 
pp. 691-698 (1998). 

[0038] Thus, this approach requires an overlap between adjacent 
fields of view, as illustrated in Figs 3 and 4. Typical overlaps 
15 are on the order of 10% of the tile size. 

[0039] It is noted that typical optical systems used in imaging 
produce an inverted image; that is, the orientation of the x and 
y axes of the object are opposite in sample surface and in the 

20 image. Therefore, in their raw form these images cannot be used 
to construct a composite image. Rather, before either 
concatenation or stitching of the various tiles is carried out, 
each image needs to be inverted to match the orientation of the 
object. This operation can be done in conventional manner either 

25 in software or hardware. 
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[0040] Figs. 5, 6, 7 and 8 illustrate the positive results 
produced by the geometric-calibration procedure of the invention. 
Fig. 5 shows the image produced by scanning a sample area with a 
5 two-microscope array (the example is limited to two tiles for 
simplicity) in step-and-repeat mode (only two frames are 
illustrated) without correction for distortion. The distortion 
error introduced by the right-hand microscope is illustrated in 
dashed line next to the ideal distortion-free fields of view 

10 characterized by the overlapping solid-line rectangles. The 

distortion is more clearly visible in the enlarged view of Fig. 
6, which corresponds to a section of the overlap region wherein 
the distortion error produces vertical discontinuities in the 
data. Fig. 7 is the image produced by scanning the same sample 

15 without correction for geometric misalignments between the two 
microscopes. The angular error introduced by the right-hand 
microscope is illustrated by the corresponding tilted fields of 
view. Fig. 8 is the image produced with the same sample data 
after calibration of the array microscope for geometric 

20 uniformity, as described, and the subsequent application of the 

resulting correction factors to the raw image data. In all cases 
the checkerboard images corresponding to each acquisition frame 
were aligned and stitched together applying the maximum cross- 
correlation algorithm to a single pair of adjacent image tiles. 
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[0041] According to another aspect of the invention, the device 
is calibrated to establish a uniform spectral response across the 
array microscope. That is, correction factors are generated to 
normalize the spectral response of each detector in the array. 
5 When images belonging to different fields of view are acquired 
using separate detectors and/or light sources, there is a 
possibility of variation in the spectral responses obtained from 
the various detectors. These differences may stem, for example, 
from light sources illuminating different fields of view at 

10 different temperatures, or from different ages of the light 
bulbs, or from different filter characteristics, etc. These 
differences also need to be addressed and normalized in order to 
produce a composite image of uniform quality, especially when the 
images are subject to subsequent computer analysis, such as 

15 described in U.S. Patent No. 6,404,916. Similar differences may 
be present in the spectral response of the detectors as a result 
of variations in the manufacturing process or coating properties 
of the various detectors. 

20 [0042] A suitable calibration procedure for spectral response to 
establish correction factors for each field of view may be 
performed, for example, by measuring the response to a set of 
predefined target signals, such as calibrated illumination 
through color filters. For each field of view the response to 
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red, green and blue channels can be calculated using any one of 
several prior-art methods, such as described in W. Gross et al., 
"Correctability and Long-Term Stability of Infrared Focal Plane 
Arrays", Optical Engineering, Vol. 38(5), pp. 862-869, May 1999; 
5 and in A. Fiedenberg et al., "Nonunif ormity Two-Point Linear 
Correction Errors in Infrared Focal Plane Arrays," Optical 
Engineering, Vol. 37(4), pp. 1251-1253, April 1998. The images 
acquired from the system can then be corrected for any 
non-uniformity across an individual field of view or across the 

10 entire array. As one skilled in the art would readily 

understand, correction factors may be implemented in the form of 
look-up tables or correction curves applied to the acquired 
images. The correction for differences in the spectral response 
can be carried out on the fly through computation during data 

15 acquisition, such as by using a programmable hardware device. 

Alternatively, the correction may be implemented structurally by 
modifying the light-source/detector optical path to produce the 
required compensation (for example, by inserting correction 
filters, changing the temperature of the light source, etc.). 

20 

[0043] It is understood that all these procedures aim at 
producing a uniform spectral response in each acquisition system, 
such that no variation in the image characteristics is produced 
as a result of device non-uniformities across the entire 
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composite image. Therefore, in case when such a corrective 
procedure is not carried out prior to the formation of the 
composite image, the spectral-response characteristics of each 
field of view should be used in post-imaging analysis to 
compensate for differences. As one skilled in the art would 
readily understand, these corrections can also be applied to a 
detector as a whole or on a pixel-by-pixel basis. 

[0044] Figs. 9 and 10 illustrate the positive results produced by 
the spectral-response calibration procedure of the invention. 
Fig. 9 is an image produced by scanning a sample area with a two- 
microscope array in step-and-repeat mode without correction for 
differences in spectral response. The figure shows that the left 
and right microscopes exhibit a relatively higher spectral 
response to red and green, respectively (which produce darker and 
lighter pictures, respectively, in black and white) . Fig. 10 is 
the corresponding image produced with the same data after 
calibration of the array for spectral-response uniformity, as 
described, and the subsequent application of the resulting 
correction factors to the raw image data. The correction factors 
were calculated for each pixel of each field of view in response 
to red, green and blue channels using the methods described in W. 
Gross et al. and A. Fiedenberg et al., supra. 
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[0045] According to yet another aspect of the invention, the 
device is calibrated to establish a uniform gain and offset 
response across the array microscope. Because of variations in 
the currents supplied to the light sources to the various optical 
5 systems , in the optical properties of the systems, in the 

detector pixel responses, in the temperatures of the sensors, 
etc., the combined response of the instrument to light may vary 
from pixel to pixel and from one field of view to another. Such 
variations are manifested in the composite image as sections with 

10 different properties (such as brightness and contrast) . In 

addition, such non-uniform images may cause different responses 
to each field of view to be obtained when automated analysis 
tools are used. Therefore, it is important that these variations 
also be accounted for by calibration, which can be achieved by 

15 measuring the response produced by a know target to generate a 

set of gain and offset coefficients. For example, a known target 
is placed in the field of view of each optical system (such a 
target could be a neutral-density filter with known optical 
density) . A series of images is taken for different optical 

20 density values. Based on these measurements, the gain and offset 
of each pixel in each field of view are calculated using one of 
several well known procedures, such as outlined in W. Gross et 
al. and A. Fiedenberg et al., supra. Appropriate correction 
coefficients are then computed to normalize the image properties 
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of each pixel (or on average for a field of view) so that the 
same gain/offset response is measured across the entire set of 
fields of view. A single target gain and a single target offset 
may be used to normalize the response of the detector/light- 
5 source combination at two signal levels and, assuming linear 

behavior, the resulting correction factors may be used between 
those levels. Correction factors for additional linear segments 
of signal levels may be similarly computed, if necessary, to 
cover a greater signal intensity span. 

10 

[0046] Figs. 11 and 12 illustrate the improvements produced by 
the gain/offset calibration procedure of the invention. Fig. 11 
is an image produced by scanning a uniform-background sample area 
with the same array microscope in step-and-repeat mode without 

15 correction for differences in detector gain and offset. The 
difference in the intensity registered by adjacent lines of 
pixels clearly illustrates non-uniformity in the response of the 
various detector pixels. Fig. 12 is the corresponding image 
produced with the same data after calibration of the array for 

20 gain and offset uniformity according to the invention, as 
described, and the subsequent application of the resulting 
correction coefficients to the raw image data. The correction 
coefficients were calculated for each pixel of each field of view 
using the procedure described in W. Gross et al. and A. 
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Fiedenberg et al., supra. 

[0047] Obviously, all three kinds of corrections described herein 
may be, and preferably are, implemented at the same time on the 
5 image data acquired to produce a composite image. To the extent 
that normalizing corrections are implemented through linear 
transformations, a cumulative matrix of coefficients can be 
calculated and used to effect one, two or . all three kinds of 
corrections. In addition, as mentioned above, a composite image 

10 can be constructed using either a concatenation or a stitching 
technique. The first method is preferred because of its speed, 
but it is also much more difficult to implement because the exact 
position of each tile in the patchwork of images (or 
checkerboards) acquired in successive frames with an array 

15 microscope needs to be known with an accuracy better than the 

sampling distance. Thus, in order to improve the knowledge about 
the relative position of each field of view at each frame, the 
image acquisition is preferably carried out at the same instant 
for all detectors in the array microscope device. This requires 

20 a means of synchronization of all detectors in the system. One 
approach is to use one of the detectors as a master and the rest 
of detectors as slaves. Another approach is to use an external 
synchronization signal, such as one coupled to a position sensor 
for the stage, or a signal produced by stroboscopic illumination, 
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or one synchronized to the light source. 

[0048] Alternatively, a less precise knowledge about the position 
of each field of view can be combined with conventional stitching 
5 techniques to construction of the composite image. Each 

checkerboard of images acquired simultaneously at each frame can 
be used as a ^single image' because the geometric relationship 
between the images is preserved during the stitching process. 
Thus, each checkerboard frame is seamlessly fused with adjacent 

10 ones in the composite image simply by applying conventional 

stitching (such as correlation techniques) to a single pair of 
adjacent images, knowing that the remaining images remain in 
fixed relation to them. Such technique can significantly speed 
up the process of overall image construction when the exact 

15 position of each checkerboard in the composite image is not 
known. 

[0049] The procedure herein disclosed gives good results for 
objects that are flat within the depth of field of the individual 
20 optical systems. For objects that extend beyond such depth of 
field, additional refocusing may be required. This can 
be done most conveniently using an array microscope where each of 
the optical systems can be focused independently. Another way to 
compensate for variations in object height is to use a cubic 
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phase plate, as described in U.S. Patent No. 6,069,738. 

[0050] Thus, a method has been disclosed to produce a seamless 
color composite image of a large object area by acquiring data in 
5 step-and-repeat fashion with an array microscope. The method 

teaches the concept of normalizing all individual microscopes to 
produce images corrected for spatial misalignments and having 
uniform spectral-response, gain, offset, and aberration 
characteristics . 

10 

[0051] It is noted that the invention has been described in 
terms of an array microscope adapted to scan in step-and-repeat 
fashion, but the same need to produce a uniform composite image 
exists when the array microscope is used in a linear scan, as 

15 described in PCT/US02/08286 . In such cases, a linear array of 
miniaturized microscopes is preferably provided with adjacent 
fields of view that span across a first dimension of the object, 
and the object is translated past the fields of view across a 
second dimension to image the entire object. Because each 

20 miniaturized microscope is larger than its field of view, the 

individual microscopes of the imaging array are staggered in the 
direction of scanning so that their relatively smaller fields of 
view are offset over the second dimension but aligned over the 
first dimension. Thus, the detector array provides an 
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effectively continuous linear coverage along the first dimension, 
which eliminates the need for mechanical translation of the 
microscope in that direction and providing a highly advantageous 
increase in imaging speed by permitting complete coverage of the 
5 sample surface with a single scanning pass along the second 
dimension. Inasmuch as a composite picture is created by 
combining swaths measured by individual microscopes and 
associated detectors, though, the same critical need exists for 
uniformity in the characteristics of the images acquired across 
10 the array. 

[0052] Therefore, while the invention has been shown and 
described herein in what is believed to be the most practical and 
preferred embodiments with reference to array microscopes 

15 operating in step-and-repeat scanning mode, it is recognized that 
it is similarly applicable to linear scanning. Accordingly, it 
is understood that departures can be made within the scope of the 
invention, which is not to be limited to the details disclosed 
herein but is to be accorded the full scope of the claims so as 

20 to embrace any and all equivalent methods and products. 
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